Dynomotion

Group: DynoMotion Message: 10181 From: moray_cuthill Date: 9/17/2014
Subject: Improving my EStop/Toolchange monitoring

Hi,

 

after a missed tool change today, I ended up crashing my lathe, so I'm looking to improve my toolchange error detection. This isn't the first time motion has started without a locked turret, and I think the basic problem lies with Mach3 not responding correctly.

 

The toolchange sequence is handled by a Click PLC with position details sent via Modbus, along with a TCALLOW (Toolchange allowed) output from the KFlop, and a TCOK signal input to the KFlop, with Mach3 being used as the interface.

 

The basic process when a toolchange is requested is -

Mach activates TCALLOW to let the PLC know that it is allowed to change tool.

The new tool position is transferred via a user/oem dro and Modbus.

At this point the TCOK signal goes inactive because the requested position no longer matches the actual position.

The PLC carries out the tool change sequence, and once the requested position matches the actual position and the turret is locked, does the PLC set the TCOK signal active.

 

 

For EStop monitoring, my init.c program uses a SYSTEMOK bit that gets set during initialisation, then within an infinite loop, there are various tests to check/test system status and if any check/test fails, SYSTEMOK gets cleared, which then triggers an ESTOP within Mach along with disabling all channels.

SYSTEMOK can only be reset by re-running the init.c, i.e. once it's cleared, there is nothing within the infinite loop that can reset it.

 

My test for the toolchanger is simply-

if(!ReadBit(TCOK) && ReadBit(TCACTIVE)) { // we need to handle TC seperate, as we lose TCOK during a change, so as long as TCActive is set at the same time, we have nothing to worry about
   ClearBit(SYSTEMOK);
  }

 

Now when the crash occured, the TCOK signal was inactive and the TCALLOW still active, which means that because Mach thought the toolchange went OK, it started sending motion commands again, and because TCALLOW was still active, the init.c test failed to notice any issues, resulting in lots of sparks.

Mach 3 shouldn't exit the toolchange script until TCOK is active, but for some reason it does, and I have noticed very occasionally the coolant (only other output controlled via Mach3) doesn't switch of, so I suspect there is an issue with inputs/outputs being updated correctly. I'm not sure what software versions the lathe is running, as I've not updated it since the start of the year.

 

Regardless, I'd like to make the safety checks more robust.

I'm thinking I need to add something that either prevents X/Y motion when TCOK is not active, and/or triggers an E-Stop when TCOK is lost and X/Y axis are moving, however I'm not sure what commands, if any, are avaiable for monitoring/testing for movement.

Any suggestions?

 

Thanks

Moray

 

Group: DynoMotion Message: 10182 From: Tom Kerekes Date: 9/17/2014
Subject: Re: Improving my EStop/Toolchange monitoring
Hi Moray,

Sorry to hear about the crash.

I have a question.  How does your Mach3 code know when it is a valid time to start checking TCOK?   If I understand you correctly all the Mach3 code does is send the new tool position which will eventually make its way to the PLC and then the PLC will set TCOK false.  But then there will be an additional delay before TCOK goes false to Mach3. If the Mach3 code checks TCOK too soon it can see it as true and might thing the operation is complete when it actually isn't.  Have you considered this?

Regards
TK

Group: DynoMotion Message: 10183 From: Moray Cuthill Date: 9/18/2014
Subject: Re: Improving my EStop/Toolchange monitoring
Hi Tom,
 
I forgot to mention that bit.
After Mach has sent the new position, it enters a while loop that waits for TCOK to go inactive, then another while loop that then waits for TCOK to active, after which the M6 script exits. I never used delays, as a single position move can take under a second, so given Mach's 10hz macro update, I didn't see any benefit to using a delay.
 
There are two things that do happen very occasionally.
The first is as I mentioned for the coolant, the TCALLOW stays active, which if it's already active when a tool change starts, everything just hangs waiting. This happens maybe once every few hundred tool changes.
The second, which only happens maybe once every few thousand tool changes, is the toolchanger actually fails to lock or misses position.
The PLC is programmed so only one move attempt is made each time TCALLOW goes active, and also to stop turret movement if for any reason it continues to rotate for more than a few seconds.
The only way to get around the above problems is to manually cycle the TCALLOW output within KMotion.
 
What I suspect may be happening, is the turret does briefly lock long enough for TCOK to become active again satisfying Mach, but then the turret unlocks, and because it just so happens to occur when TCALLOW has failed to deactivate, the KFlop safety check thinks everything is ok, and nothing notices we have a problem.
 
I have thought about some kind of time delay monitor for TCALLOW so at least things get stopped eventually, but I'd prefer some kind of more robust check so things get stopped quicker.
 
Thanks
Moray

On Thu, Sep 18, 2014 at 5:51 AM, Tom Kerekes tk@... [DynoMotion] <DynoMotion@yahoogroups.com> wrote:
 

Hi Moray,

Sorry to hear about the crash.

I have a question.  How does your Mach3 code know when it is a valid time to start checking TCOK?   If I understand you correctly all the Mach3 code does is send the new tool position which will eventually make its way to the PLC and then the PLC will set TCOK false.  But then there will be an additional delay before TCOK goes false to Mach3. If the Mach3 code checks TCOK too soon it can see it as true and might thing the operation is complete when it actually isn't.  Have you considered this?

Regards
TK

Group: DynoMotion Message: 10184 From: Dan Date: 9/18/2014
Subject: Re: Improving my EStop/Toolchange monitoring
Attachments :
    Moray,
    Two things come to mind reading about your problem.
     
    1. a limit switch or sensor that detects the tool changer is locked and ready to go. Does it have one? If so how is it being used?
     
    2. A one shot handshake between the PLC and Mach3. So once the tool changer is done the PLC sets a bit high “PLChandShake” for mach3 to read, additionally that bit locks out any actions in the turret when its active. Once mach3 has read it then it sends back a complimentary handshake bit “Mach3HandShake” saying it is going to run again. Once the PLC gets that gets the “Mach3HandShake” it clears the “PLChandShake” bit, but the “Mach3HandShake” would also lock out any actions in the turret (both bit parallel). 
     
    The advantage to this is, the “ok go” signal to mach3 is in the off state at the start of a tool change so mach3 cant accidently read the pre-tool change ok as a post tool change ok.  I would go as far as to program it to inspect for both conditions.
     
    Event timings would be something like this. The 45* lines should be instant offs not decaying off’s, this is all I could get excel to do for me this morning.
     
    Mach 3 Handshake
     
     
    Dan
     
    Sent: Thursday, September 18, 2014 4:03 AM
    Subject: Re: [DynoMotion] Improving my EStop/Toolchange monitoring
     
     

    Hi Tom,
     
    I forgot to mention that bit.
    After Mach has sent the new position, it enters a while loop that waits for TCOK to go inactive, then another while loop that then waits for TCOK to active, after which the M6 script exits. I never used delays, as a single position move can take under a second, so given Mach's 10hz macro update, I didn't see any benefit to using a delay.
     
    There are two things that do happen very occasionally.
    The first is as I mentioned for the coolant, the TCALLOW stays active, which if it's already active when a tool change starts, everything just hangs waiting. This happens maybe once every few hundred tool changes.
    The second, which only happens maybe once every few thousand tool changes, is the toolchanger actually fails to lock or misses position.
    The PLC is programmed so only one move attempt is made each time TCALLOW goes active, and also to stop turret movement if for any reason it continues to rotate for more than a few seconds.
    The only way to get around the above problems is to manually cycle the TCALLOW output within KMotion.
     
    What I suspect may be happening, is the turret does briefly lock long enough for TCOK to become active again satisfying Mach, but then the turret unlocks, and because it just so happens to occur when TCALLOW has failed to deactivate, the KFlop safety check thinks everything is ok, and nothing notices we have a problem.
     
    I have thought about some kind of time delay monitor for TCALLOW so at least things get stopped eventually, but I'd prefer some kind of more robust check so things get stopped quicker.
     
    Thanks
    Moray
     
    On Thu, Sep 18, 2014 at 5:51 AM, Tom Kerekes tk@... [DynoMotion] <DynoMotion@yahoogroups.com> wrote:
     

    Hi Moray,

    Sorry to hear about the crash.

    I have a question.  How does your Mach3 code know when it is a valid time to start checking TCOK?   If I understand you correctly all the Mach3 code does is send the new tool position which will eventually make its way to the PLC and then the PLC will set TCOK false.  But then there will be an additional delay before TCOK goes false to Mach3. If the Mach3 code checks TCOK too soon it can see it as true and might thing the operation is complete when it actually isn't.  Have you considered this?

    Regards
    TK
     
    Group: DynoMotion Message: 10188 From: Tom Kerekes Date: 9/18/2014
    Subject: Re: Improving my EStop/Toolchange monitoring
    Hi Moray,

    I wasn't suggesting adding a delay.  I was pointing out that there will be an indeterminate delay from any signal from the PLC to Mach3 because of Windows/PC/MODBUS/USB.

    I think there is still a possible flaw in your technique where Mach3 could miss TCOK ever going low.  But I think that would result in a hang, not a crash.

    I find the classic way to handle this type of thing (which I believe is essentially the way Dan describes) is to use a "two wire handshake". 

    #1 One side puts a signal high,
    #2 when the other side sees it high it places a return signal high
       (tool change performed here)
    #3 when the first side sees the return signal high it puts its signal low
    #4 when the other side sees the signal low it puts its return signal low
    #5 when the first side sees the return signal low it continues on

    This technique has no "race conditions" or sample rate requirements of either side.

    I can't see any reason why setting an Output would not always work.

    I don't really understand your "watchdog" idea to prevent a crash.  But there is example code in KStep\InitKStep3Axis.c that watches for any motion and immediately enables KSTEP.  You might make use of something like that.

    Regards
    TK




    From: "Moray Cuthill moray.cuthill@... [DynoMotion]" <DynoMotion@yahoogroups.com>
    To: DynoMotion@yahoogroups.com
    Sent: Thursday, September 18, 2014 2:03 AM
    Subject: Re: [DynoMotion] Improving my EStop/Toolchange monitoring

     
    Hi Tom,
     
    I forgot to mention that bit.
    After Mach has sent the new position, it enters a while loop that waits for TCOK to go inactive, then another while loop that then waits for TCOK to active, after which the M6 script exits. I never used delays, as a single position move can take under a second, so given Mach's 10hz macro update, I didn't see any benefit to using a delay.
     
    There are two things that do happen very occasionally.
    The first is as I mentioned for the coolant, the TCALLOW stays active, which if it's already active when a tool change starts, everything just hangs waiting. This happens maybe once every few hundred tool changes.
    The second, which only happens maybe once every few thousand tool changes, is the toolchanger actually fails to lock or misses position.
    The PLC is programmed so only one move attempt is made each time TCALLOW goes active, and also to stop turret movement if for any reason it continues to rotate for more than a few seconds.
    The only way to get around the above problems is to manually cycle the TCALLOW output within KMotion.
     
    What I suspect may be happening, is the turret does briefly lock long enough for TCOK to become active again satisfying Mach, but then the turret unlocks, and because it just so happens to occur when TCALLOW has failed to deactivate, the KFlop safety check thinks everything is ok, and nothing notices we have a problem.
     
    I have thought about some kind of time delay monitor for TCALLOW so at least things get stopped eventually, but I'd prefer some kind of more robust check so things get stopped quicker.
     
    Thanks
    Moray

    On Thu, Sep 18, 2014 at 5:51 AM, Tom Kerekes tk@... [DynoMotion] <DynoMotion@yahoogroups.com> wrote:
     
    Hi Moray,

    Sorry to hear about the crash.

    I have a question.  How does your Mach3 code know when it is a valid time to start checking TCOK?   If I understand you correctly all the Mach3 code does is send the new tool position which will eventually make its way to the PLC and then the PLC will set TCOK false.  But then there will be an additional delay before TCOK goes false to Mach3. If the Mach3 code checks TCOK too soon it can see it as true and might thing the operation is complete when it actually isn't.  Have you considered this?

    Regards
    TK


    From: "moray.cuthill@... [DynoMotion]" <DynoMotion@yahoogroups.com>
    To: DynoMotion@yahoogroups.com
    Sent: Wednesday, September 17, 2014 5:09 PM
    Subject: [DynoMotion] Improving my EStop/Toolchange monitoring

     
    Hi,
     
    after a missed tool change today, I ended up crashing my lathe, so I'm looking to improve my toolchange error detection. This isn't the first time motion has started without a locked turret, and I think the basic problem lies with Mach3 not responding correctly.
     
    The toolchange sequence is handled by a Click PLC with position details sent via Modbus, along with a TCALLOW (Toolchange allowed) output from the KFlop, and a TCOK signal input to the KFlop, with Mach3 being used as the interface.
     
    The basic process when a toolchange is requested is -
    Mach activates TCALLOW to let the PLC know that it is allowed to change tool.
    The new tool position is transferred via a user/oem dro and Modbus.
    At this point the TCOK signal goes inactive because the requested position no longer matches the actual position.
    The PLC carries out the tool change sequence, and once the requested position matches the actual position and the turret is locked, does the PLC set the TCOK signal active.
     
     
    For EStop monitoring, my init.c program uses a SYSTEMOK bit that gets set during initialisation, then within an infinite loop, there are various tests to check/test system status and if any check/test fails, SYSTEMOK gets cleared, which then triggers an ESTOP within Mach along with disabling all channels.
    SYSTEMOK can only be reset by re-running the init.c, i.e. once it's cleared, there is nothing within the infinite loop that can reset it.
     
    My test for the toolchanger is simply-
    if(!ReadBit(TCOK) && ReadBit(TCACTIVE)) { // we need to handle TC seperate, as we lose TCOK during a change, so as long as TCActive is set at the same time, we have nothing to worry about
       ClearBit(SYSTEMOK);
      }
     
    Now when the crash occured, the TCOK signal was inactive and the TCALLOW still active, which means that because Mach thought the toolchange went OK, it started sending motion commands again, and because TCALLOW was still active, the init.c test failed to notice any issues, resulting in lots of sparks.
    Mach 3 shouldn't exit the toolchange script until TCOK is active, but for some reason it does, and I have noticed very occasionally the coolant (only other output controlled via Mach3) doesn't switch of, so I suspect there is an issue with inputs/outputs being updated correctly. I'm not sure what software versions the lathe is running, as I've not updated it since the start of the year.
     
    Regardless, I'd like to make the safety checks more robust.
    I'm thinking I need to add something that either prevents X/Y motion when TCOK is not active, and/or triggers an E-Stop when TCOK is lost and X/Y axis are moving, however I'm not sure what commands, if any, are avaiable for monitoring/testing for movement.
    Any suggestions?
     
    Thanks
    Moray
     





    Group: DynoMotion Message: 10189 From: Moray Cuthill Date: 9/18/2014
    Subject: Re: Improving my EStop/Toolchange monitoring
    Attachments :
      Hi Dan and Tom,
       
      thanks for the replies.
      To answer Dan first, the turret has a locked signal which is connected to the PLC. The PLC monitors and controls everything to do with the turret, and only activates TCOK provided the turret is on position (it continually compares the actual position against the requested position) and locked. If the turret should unlock for any reason, or not be on position, the TCOK signal is deactivated.
      My main reason for the PLC was lack of inputs on the Kanalog (I bought the PLC only weeks before the Konnect became available!). All 8 are used, so I rely on the TCOK to tell me everything with the turret is good. The only time TCOK should go inactive is during a toolchange.
       
      And having just checked over the PLC ladder, I do have a flaw whereby an unlock situation won't disable the TCOK signal (essentially as part of the turret position checking, I check the lock first, which if unlocked resets a turret position valid bit, however the next rung checks to make sure the turret position is within range, so sets the bit valid again provided the turret encoder is reporting a valid position, which it will unless the encoder happens to go faulty)
       
      Having just altered the PLC ladder to fix that, it should help improve reliability.
       
       
      Tom,
      the issue with outputs not being activated/deactivated happens reasonably often. It's most notable with the coolant, as a cycle will often finish and the coolant will still be running. When this happens, Mach shows the coolant as off, and the coolant has to be toggled on/off for it to physically turn off.
      I also get the occasional hang during a toolchange, which I suspect is the same issue, as Mach will hang, and when the TCALLOW bit is checked using KMotion, it shows as being active. Toggling the bit off/on allows the PLC to start the toolchange process (as I mentioned in an earlier email, the PLC is coded so it only attempts one toolchange per TCALLOW activation), and everything continues as usual.
      Those are the only two outputs which are directly controlled via Mach, so it's the only items I ever notice an issue. There is a MachEnabled virtual bit, however it only gets controlled when the reset button is used, so I'm not as likely to notice an issues with that.
       
      I also checked earlier, and the lathe is running 4.31s. Other than these couple issues, it's been running well, and I like to not touch things when they're working.
       
      I had forgotten about that code in the KStep example. That looks like a good base for an additional watchdog.
       
      Thanks
      Moray
       

      On Thu, Sep 18, 2014 at 2:24 PM, 'Dan' engnerdan@... [DynoMotion] <DynoMotion@yahoogroups.com> wrote:
       

      Moray,
      Two things come to mind reading about your problem.
       
      1. a limit switch or sensor that detects the tool changer is locked and ready to go. Does it have one? If so how is it being used?
       
      2. A one shot handshake between the PLC and Mach3. So once the tool changer is done the PLC sets a bit high “PLChandShake” for mach3 to read, additionally that bit locks out any actions in the turret when its active. Once mach3 has read it then it sends back a complimentary handshake bit “Mach3HandShake” saying it is going to run again. Once the PLC gets that gets the “Mach3HandShake” it clears the “PLChandShake” bit, but the “Mach3HandShake” would also lock out any actions in the turret (both bit parallel). 
       
      The advantage to this is, the “ok go” signal to mach3 is in the off state at the start of a tool change so mach3 cant accidently read the pre-tool change ok as a post tool change ok.  I would go as far as to program it to inspect for both conditions.
       
      Event timings would be something like this. The 45* lines should be instant offs not decaying off’s, this is all I could get excel to do for me this morning.
       
      Mach 3 Handshake
       
       
      Dan
       
      Sent: Thursday, September 18, 2014 4:03 AM
      Subject: Re: [DynoMotion] Improving my EStop/Toolchange monitoring
       
       

      Hi Tom,
       
      I forgot to mention that bit.
      After Mach has sent the new position, it enters a while loop that waits for TCOK to go inactive, then another while loop that then waits for TCOK to active, after which the M6 script exits. I never used delays, as a single position move can take under a second, so given Mach's 10hz macro update, I didn't see any benefit to using a delay.
       
      There are two things that do happen very occasionally.
      The first is as I mentioned for the coolant, the TCALLOW stays active, which if it's already active when a tool change starts, everything just hangs waiting. This happens maybe once every few hundred tool changes.
      The second, which only happens maybe once every few thousand tool changes, is the toolchanger actually fails to lock or misses position.
      The PLC is programmed so only one move attempt is made each time TCALLOW goes active, and also to stop turret movement if for any reason it continues to rotate for more than a few seconds.
      The only way to get around the above problems is to manually cycle the TCALLOW output within KMotion.
       
      What I suspect may be happening, is the turret does briefly lock long enough for TCOK to become active again satisfying Mach, but then the turret unlocks, and because it just so happens to occur when TCALLOW has failed to deactivate, the KFlop safety check thinks everything is ok, and nothing notices we have a problem.
       
      I have thought about some kind of time delay monitor for TCALLOW so at least things get stopped eventually, but I'd prefer some kind of more robust check so things get stopped quicker.
       
      Thanks
      Moray
       
      On Thu, Sep 18, 2014 at 5:51 AM, Tom Kerekes tk@... [DynoMotion] <DynoMotion@yahoogroups.com> wrote:
       

      Hi Moray,

      Sorry to hear about the crash.

      I have a question.  How does your Mach3 code know when it is a valid time to start checking TCOK?   If I understand you correctly all the Mach3 code does is send the new tool position which will eventually make its way to the PLC and then the PLC will set TCOK false.  But then there will be an additional delay before TCOK goes false to Mach3. If the Mach3 code checks TCOK too soon it can see it as true and might thing the operation is complete when it actually isn't.  Have you considered this?

      Regards
      TK